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ABSTRACT 


Organizations are becoming increasingly aware that learning can be a source of 
competitive advantage. The United States Navy is not immune to this realization and has 
established the Navy Lessons Learned System (NLLS) as the singular Navy program for 
the collection, validation, and distribution of unit feedback. NLLS provides naval 
personnel a means to share observations, document deficiencies, convey solutions, and 
innovate tactics, techniques, or procedures (TTP). The purpose of this thesis is to 
examine the various factors that influence organizational learning, such as structure, 
environment, and culture, and to examine how Information Technology can be used to 
support or enhance organizational learning in the Navy. The research concludes that 
NLLS has improved organizational learning but has not attained as widespread use as is 
possible. Recommendations are provided to improve the program as well as increase 
NLLS exposure to the fleet and to the potential users of the system. 
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I. INTRODUCTION 


Chris Argyris and Donald A. Schon, in their classic book on organizational 
learning, define it as the process of “detection and correction of errors.” In their view, 
organizations learn through the individuals who act as agents for them: “The individuals’ 
learning activities in turn, are facilitated or inhibited by an ecological system of factors 
that may be called an organization learning system.” (Argyris and Schon, 1978) George 
Huber expands on this, stating, “An organization learns when, through its processing of 
information, the range or likelihood of its potential behaviors is changed.” (Huber, 1991) 
Offering us a procedural definition are other noted authors, Raanan Lipshitz and 
Micha Popper who define a learning organization as: 

A learning organization is an organization that institutes organization¬ 
learning mechanisms and operates them regularly. Organization Learning 
Mechanisms are defined as institutionalized structural and procedural 
arrangements that allow organizations to systematically collect, analyze, 
store, disseminate, and use information that is relevant to the effectiveness 
of the organization. (Lipshitz and Popper, 1996) 

Organizations as diverse as Motorola, BP Amoco, and the United States Army are 
becoming increasingly aware of the concept of organizational learning in order to acquire 
a competitive advantage, increase their capacity to innovate, or improve effectiveness. 

With this need in mind, every day around the world naval units are involved in 
either real world exercises or training missions that mirror the real world. Traditionally, 
emphasis is placed on the preparation-through-execution phase. However, if we do not 
review, analyze, and disseminate the exercises and missions we have just concluded, we 
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lose potentially the most important aspect: the lessons learned, and with them our ability 
as an organization to learn. While this may seem a simple enough concept, actualizing it 
requires two major elements. First, a way to collect and record these valuable 
contributions, and second, a way to share the information throughout the organization. 
Without both we are destined to repeat past mistakes. 

A. PURPOSE 

This research evaluates the impact that the implementation of an Information 
Technology infrastructure has had on the potential for organizational learning through the 
Navy Lessons Learned System (NLLS). The purpose of this study is to explore and 
discuss the influence that technology, especially information technology, has had on the 
organizational learning process in the Navy. 

The primary research objective is to explore the answer to the following question: 
Can the use of Information Technology in the Navy’s Lessons Learned System be used to 
improve organizational learning? Secondary questions include: How effective is the 
Navy Lessons Learned System as a tool for promoting organizational learning? Is 
Information Technology being used effectively to analyze collected information to 
identify a recurring weakness in tactics, techniques, or procedures? And lastly, to what 
degree has Information Technology improved the process or availability of disseminating 
relevant information to potential users? 

This research targets the information systems at the Center for Navy Lessons 
Learned. A secondary purpose of this thesis is to provide the Naval Warfare and 


2 




Development Center with recommendations for improving and enhancing the existing 
organizational learning process as evaluated against that of other learning organizations. 

B. BACKGROUND 

Today, in an era of declining defense dollars, the Department of Defense and the 
United States Navy face continually increasing pressures to reduce spending. They are 
responding by reducing the numbers of ships and aircraft, downsizing military personnel, 
and decreasing training opportunities for those who stay. To surmount the budget 
challenges, the military must reap the maximum benefit from the opportunities still 
available. 

The overriding objective of the Navy’s Lessons Learned program is to provide a 
fleet-wide benefit from operational exercises and experiences. To be successful in this 
endeavor, the program must create a coordinated process for capturing, securing, 
retrieving, and distributing the organization’s knowledge. 

A common phrase heard in the business world is, “If only we knew what we 
know, we would be twice as profitable.” But why is there a need for the Navy to 
encourage knowledge sharing? We do not contend in the same competitive marketplaces 
as corporate America; nor is the United States currently at war with anyone. However, 
like all government organizations, today’s military is competing for scarce resources and 
must take full advantage of the opportunities presented. Consequently, we have seen 
lessons learned in training exercises that often must be relearned in conflict, sometimes 
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with fatal results. A GAO study published in 1994 points out one such example that 
occurred during the recent military experience in the Persian Gulf: 

“During Operation Desert Shield in 1990-1991, Jive Army protective 
bunker collapses caused three fatalities and three other injuries. The 
cause sited in a subsequent lessons learned report was that ...locally 
available materials were used to construct bunkers deviating from 
standard design. ” (GAO, 1994) 

In 1988 the Center for Army Lessons Learned (CALL) had reports that units at the 
combat training centers had difficulty in obtaining support materials for their protective 
bunkers, which caused them to use locally available materials. The results were “poorly 
constructed bunkers made of non-standard design and materials.” This prompted a 
lessons learned report that was disseminated by CALL in 1988. Because of “inadequate 
follow-up and corrective action,” a similar lessons learned report had to be generated in 
1991, this time after a lesson with far more severe consequences. (GAO, 1994) 

C. SCOPE AND METHODOLOGY 

The scope of this study focuses on the use of the Navy Lessons Learned System 
(NLLS) as a structured method for organizational learning. The primary concentration of 
the study is the use of organizational learning constructs and models as a framework for 
comparing procedures in place at the Naval Warfare and Doctrine Command and at least 
one other military organization. Current IT policies are examined to ascertain if 
recommendations or conclusions can be made to aid in improving the existing IT 
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infrastructure at NLLS. The goal is to increase NLLS ability to provide the best 
relationship between the information system organization and the customers they support. 

D. RESEARCH METHODOLOGY 

The research techniques used for this thesis comprise the following steps: 

1. Conduct a thorough literature search of the NLLS mission, purpose, history, 
and organizational structure from documents provided by the Navy Warfare Development 
Command. 

2. Gather technical information about the IT infrastructure and collection process 
by conducting telephone interviews with relevant members of the organization. 

3. Conduct a literature search of Naval Instructions, Naval Programs, professional 
journals, books, magazine articles, and other library informational resources about the 
Navy Lessons Learned System. 

4. Conduct a literature review of different learning organizational design models 
to explore and discuss the uses of their key components for this application. 

5. Conduct a literature review of the Center for Army Lessons Learned program 
to explore and discuss possible benefits or drawbacks. 

6. Explore the ability of NLLS to deliver value to the user by employing 
emerging Information Technologies to prevent recurring weakness in training, techniques, 
or procedures. 

7. Generate a sample-based assessment of NLLS’s impact on the organization, 
using an appropriate survey. 
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E. ORGANIZATION OF STUDY 


The thesis is organized as follows. 

Chapter I Introduction : This chapter provides the purpose, background, and 
research methodology of the thesis. 

Chapter II Naval Warfare and Development Command : This chapter provides 
background information on the organization of the Navy Warfare Development 
Command, history of the Navy Lesson Learned System, methodology of collection at the 
Fleet Management sites, and validation of inputs through interaction of the Subject 
Matter Experts. 

Chapter III Organizational Learning Models : This chapter discusses individual 
learning verses organizational learning and presents the reader with several types of 
organizational learning models. The effect that structure, environment, and culture can 
have on learning is also discussed. 

Chapter IV Use of Technology in a Learning Organization : This chapter will look 
at how to build a Learning Organization and examine the use of technology to improve 
Organizational Learning. 

Chapter V Information Technology in Other Organizations : This chapter will 
provide a case study of organizational learning as it occurs in another military 
organization for comparison purposes. 

Chapter VI Findings : The chapter will present the survey questions with a 
breakdown of the responses received. 
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Chapter VII Analysis : This chapter will conduct an analysis of the results of the 


survey. 

Chapter VIII Conclusions and Recommendations . This chapter will provide 
conclusions and recommendations to improve the effectiveness of NLLS. 
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II. NAVAL WARFARE DEVELOPMENT COMMAND 


A. INTRODUCTION 


The Naval Warfare Development Command was officially established on 10 
August 1998 in Newport, Rhode Island. Its purpose is to lead warfare innovation and 
concept development, to design plan and coordinate the Navy’s Fleet Battle Experiment 
program, and to represent the Navy in Joint Experimentation. NWDC charter is to 
synchronize, approve, and disseminate Navy doctrine. 

The Navy Warfare Development Command is composed of four departments: 
Concepts Development, Maritime Battle Center, Doctrine, and Operations. (NWDC 
Public Affairs, 2000) 


Naval Warfare 
Development 
Command 




Figure 1. NWDC Organizational Chart 


1. Concepts 

The Concepts Department’s mission is to develop Navy warfighting concepts and 
identify required capabilities. These concepts are then tested and refined through Fleet 
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Battle Experiments, war games, modeling and simulations, and other processes. The 
results are expected to form the basis to develop and revise Navy Doctrine. 

By pooling ideas from many sources and taking the most promising ideas. 
Concepts can develop and identify capabilities with potential improvements to the way 
the Navy operates. These ideas are then validated through a fast-paced, six-month 
experimentation process providing for warfighting innovation. (NWDC Public Affairs, 
2000) 


2. Maritime Battle Center 

The Maritime Battle Center (MBC) was established to serve as the single point of 
contact for the Navy Fleet Battle Experimentation (FBE) and participation in Joint 
Experiments. 

The MBC is responsible for designing and planning Fleet Battle Experiments, 
coordinating the execution of these experiments in conjunction with the numbered fleet 
operational command elements, and analyzing and disseminating experiment results. The 
FBE results are then used to accelerate the delivery of innovative warfare capabilities to 
the fleet, identify concept based requirements, and evaluate new operational capabilities. 
(NWDC Public Affairs, 2000) 

3. Doctrine 

The Doctrine Department is the Navy’s primary point of contact for Naval 
Doctrine and Joint and Combined Doctrine Development. As the Fleet agent for 
operational doctrine development, the NWDC Doctrine Department facilitates fleet 
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consensus on emerging Navy and Joint doctrinal issues. (NWDCINST 5400. IX) In 
addition, the Doctrine Department has the administrative and management responsibility 
for the Navy Warfare Publication Library and the Navy Lessons Learned System. 
(NWDC Public Affairs, 2000) 

The primary means for the proposal, validation, and revision of Navy operational- 
level doctrine and inputs to Joint doctrine is through the use of a semi-annual Navy 
Doctrine Working Party (NDWP). The objectives of the NDWP are threefold: 

• Serve as a forum for Fleet engagement on current or emerging doctrinal 
issues. 

• Consolidate the Fleet input on topics at the subsequent Joint Doctrine Working 
Party. 

• Validate proposals for development/revision of Navy Warfare Publications 
(NWP). 

According to Captain Robert Nestlerode, Doctrine Department Head, proven 
operational concepts, validated through experimentation and collaborative development 
with the fleet, translate into useful doctrine. “The goal is to make the Navy’s doctrine 
development dynamic, responsive, and interactive.” (Nestlerode, 2000) 

4. Operations 

The Operations Department is the newest department at NWDC, established in 
May 1999. The purpose of the Operations Department is to provide support in the form of 
technical research, analysis, modeling and simulation, and "red cell," or adversary view, 
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of emerging concepts. In addition, the Operations Department is tasked to act as the 
NWDC implementation conduit to the Fleet Commanders in Chief, Numbered Fleet 
Commanders, Naval Systems Command, and the Integrated Warfare Requirements 
(IWAR) board in Washington D.C. (NWDC Public Affairs, 2000) 

B. OVERVIEW OF THE NAVY LESSONS LEARNED SYSTEM 

1. Navy Lessons Learned Central Site 

a. History 

The Navy Lessons Learned System (NLLS) was formally created in 1991 
by CNO direction as a result of a growing fleet interest in providing a centrally managed 
and easily accessible lessons learned database. In 1995, a Congressionally ordered GAO 
report was released that was critical of all the services and their potential to use lessons 
learned as a means to avoid repetition of past mistakes. Consequently, the Chief of Naval 
Operations improved on this system by instituting and formalizing guidance in the form 
of an OPNAV instruction. Its purpose was to standardize and delineate the NLLS 
requirement and formal feedback process. The Navy Lessons Learned System that 
developed was patterned after the Joint Chiefs of Staff (JCS) sponsored Joint Universal 
Lessons Learned System (JULLS). This system currently makes up the formal feedback 
process. (NWDC Public Affairs, 2000) 
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b. Purpose 

By definition, a lesson learned is information that contributes to the 


Navy’s corporate body of knowledge and should produce increased process efficiency and 
improved execution of future operations. Additionally, it should provide value added to 
existing Navy policy, doctrine, tactics, techniques, procedures, training, systems or 
equipment. (CINCPACFLTINST 3500.37A) 

The objective of NLLS is to provide for a Navy-wide standardized 
procedure for collection, validation, and distribution of lessons learned from fleet 
exercises and everyday operations. This provides Naval activities with a formally 
managed database of current information that has been validated by either platform or 
warfare experts in support of fleet operations. The lessons learned are submitted through 
the use of the Navy Instruction Input Program (NIIP) to the applicable commands 
operational Immediate Superior in Command (ISIC). There, they are checked for 
applicability and redundancy before being sent on to one of five Fleet Management Sites 
for validation and inclusion in the database. 

c. Structure 

The Navy Lessons Learned Central site is assigned under the Naval 
Warfare Development Command (NWDC) Doctrine department and serves as the 
centralized data collection, management, and distribution center for all NLLS inputs. The 
Central Site also is tasked to act as the Program Director and Administrator for NLLS, as 
well as provide for the quality assurance of the Navy Lessons Learned Data Base 
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(NLLSDB), and distribution of the product to fleet units. Contractor support at NWDC 
consolidates Lessons Learned, Port Visit Reports, and After Action Reports, insuring that 
they are in the proper format before releasing the data as part of the NLLSDB via a 
Secure Internet Protocol Router Network (SIPRNET) Web site and a quarterly CD-ROM 
set. 


2. Fleet Management Sites 

a. Purpose 

Fleet Management Sites (FMS) are responsible for the collection, 
processing, and validating of lessons learned submitted from their area of responsibility 
(AOR). By supplying the approval on each submission, each fleet CINC establishes 
control over the quality and validity of the lessons learned in his area before it is 
submitted to the central site for inclusion into the NLLDB. 

b. Structure 

When a fleet user submits a lesson learned, it is sent to one of five Fleet 
Management Sites, depending on the originator’s theater of operations. These sites are 
located with the numbered fleet commanders and are each manned by a civilian 
contractor senior data analyst. It is this analyst, with the help of a variety of subject matter 
experts (SMEs), who is responsible for the processing and validation of the lessons 
learned submitted by his or her subordinate commands. 
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The Fleet Management Sites are also responsible for overseeing the 
Remedial Action Program for their areas and ensuring NLLS training is conducted aboard 
each deploying command. Additionally, they conduct an annual review to verify the 
currency and quality of the database at the central site. (OPNAV3500.37B) 

3. Analysis and Validation 

a. Subject Matter Experts 

Subject Matter Experts (SMEs) are personnel in major commands, warfare 
centers, or tactical development centers who are recognized for having extensive 
knowledge and experience in a particular warfare area of operation (OPNAVINST 
3500.37B). SMEs provide the theater commanders support by reviewing and validating 
Lessons Learned and Remedial Action Program issues as required. For example, a lesson 
learned submission on Battle Group tactics in the Atlantic would go to the SMEs located 
at Tactical Training Group Atlantic for review, consideration, and validation. If it 
concerned employment of a Carrier Air Wing, it would go to the Naval Strike and Air 
Warfare Center. 

b. Remedial Action Program 

A Remedial Action Program (RAP) item is defined as a deficiency or 
shortcoming in existing policy, organization, training, education, equipment, or doctrine 
that requires actions to correct. (OPNAVINST 3500.37B) If a lesson learned submission 
meets this criterion, the Fleet Management Site will assign a RAP Working Group to this 
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item. The Management site is responsible for tracking and updating the NLLDB on the 
status of the remedial action. If an item cannot be resolved at the Management Site Level, 
the item will be forwarded to a CNO Executive Panel. Once the item is accepted, the 
CNO panel is responsible for tracking the action taken, but it is still the Management 
Site’s responsibility to update the item in the NLLDB. If the CNO Executive Panel 
rejects the item, the Management Site will note the reason in the database and cancel the 
project as appropriate. (OPNAVINST 3500.37B) 

Tactical or procedural deficiencies will be forwarded to Tactical 
Development and Evaluation (TAC D&E) Steering Committees for consideration. 
Approved considerations will be developed into new Tactics, Techniques or Procedures 
for inclusion into the Navy Warfare Publications (NWP) system. 


C. SUBMISSION, PUBLICATION, AND DISTRIBUTION 

A Fleet User will submit a Lesson Learned (LL) by means of a Navy Instructional 
Input Program (NITP) message to the Immediate Supervisor In Command (ISIC). The 
ISIC will review the LL for accuracy and forward it to the theater’s NLLS Management 
Site. The NLLS Management Sites are located as follows: 

CINCLANTFLT NLLS Management Site, Norfolk, Virginia 
CINCPACTFLT NLLS Management Site, San Diego, California 
COMUSNAVCENT NLLS Management Site, Tampa, Florida 
COMSIXTHFLT NLLS Management Site, Gaeta, Italy 
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COMSEVENTHFLT NLLS Management Site, Yokosuka, Japan 

At each Fleet Management Site (FMS) the NLLS Data Analyst has the lesson 
learned validated by Navy Staff personnel or Subject Matter Experts. Once validated, the 
FMS will make certain that the Lesson Learned Submission / Summary Report is in the 
proper format. He will then assign appropriate keywords and ensure the comments 
section uses standardized terminology. 

The validated lesson learned is forwarded to the NLLS Central Site at the Naval 
Warfare Development Command in Newport, RI, for final processing and uploading to 
the Lessons Learned SIPRNET Web Site. Along with the weekly Web site update, the 
lessons learned database is distributed quarterly on a NLLS two-volume CD-ROM. 

Lessons learned will remain in the active database until it is determined by the 
theater CINC or SMEs that they are no longer valid, have been corrected, or are no longer 
relevant to current Navy procedure. At that point, but not later than two years from the 
entry date, the lesson learned will be moved to the inactive database. 
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III. ORGANIZATIONAL LEARNING MODEL 

A. INTRODUCTION 

Organizations continually strive for improved systems of information collection, 
management, and analysis. The introduction of and rapid advancements in availability of 
computers during the last 30 years have resulted in more evolved systems for acquiring, 
storing, and disseminating information. The opportunities presented by these advances 
have produced widely varying benefits for organizations seeking to increase stakeholder 
value. This identification of knowledge as a means for competitive advantage has meant 
an exponential demand for information systems. Worldwide expenditures on information 
technology have generated significant but somewhat varying benefits to those companies 
who invest in information technology for knowledge benefits. (Drucker, 1994) This 
explosion of data often threatens to overwhelm individuals with information. Advances 
have focused on making us masters of gathering data, but not on turning that data into 
knowledge. 

Companies today gather increasing amounts of information, much of it of 
debatable utility. All too often the question asked is, “What data do you want to gather?” 
instead of. What do you plan on doing with it?” Advances in computer systems have 
made gathering and storing data simple. The problem now is finding and understanding 
the information that resides in these huge data warehouses. 

Thomas Davenport of Anderson Consulting gives a good example of this data-to- 
knowledge deficit. He writes that the CIO of a grocery chain highly regarded for its use 
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of IT once confided to him that his company analyzed at most only two percent of the 
data it collected. Another example Davenport cites is that of a midwestem grocery chain 
that finally decided to throw away its scanner data. The data “had been saved for years in 
the hope that it would someday be analyzed, but it never was.” (Davenport, 1999) 

The challenge facing today’s IT people is not to provide technology or record 
data, but to help the organization make use of this information. 

B. DATA VS. KNOWLEDGE 

Quality information is as critical a resource in the military as the ships we serve 
on. But what exactly is information, and what separates quality information from data? 
Do we know what kind of information we need, and, if so, how do we recognize, capture, 
and then pass it on to others? 

As simple as it may seem, it is important to understand that data, information, and 
knowledge are not interchangeable. Understanding the difference between the three and 
what you can and can’t get from each can mean the difference between success and 
failure in individual or organizational learning. 

1. Data 

Data can be defined as a set of discrete, objective facts about events (Davenport, 
1999). In the organizational context, it is most often described as a structured record of 
transactions. The example Davenport gives involves a trip to the local gas station. The 
receipt that prints out after the customer fills the tank consists of data: how many gallons 
the customer bought, how much he paid for each gallon, and when he bought the gas. The 
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data by itself tells the reader nothing about why the customer chose that particular gas 
station over another, when or will he come back again, or how well or poorly that gas 
station is managed. (Davenport, 1999) 

Data by itself can be evaluated in two ways, quantitatively and qualitatively. 
Quantitative measures would include how much the data costs to capture, or how long it 
takes the system to call up the data. Qualitative measures would include “timeliness, 
relevance, and clarity.” (Davenport, 1999) Do we have access to it when we need it, is it 
what we need, and can we make any sense out of it? 

Data collecting has become increasingly efficient in many organizations. But the 
problem with becoming a data culture is twofold: First, the more data you collect the 
harder it is to make sense out of it; and second, there is no inherent meaning in data. Data 
by itself tells only part of the story. 

2. Information 

One definition of information is “data that has been put into a meaningful and 
useful context and communicated to a recipient who uses it to make decisions.” (Burch 
and Grudnitski, 1986) Information must inform and, as with the transfer of any message, 
information has to have a sender and a receiver. In information, it is the receiver who 
decides what is really information. “A memo of unconnected ramblings may be 
information to the sender but to the receiver it is just noise.” (Davenport, 1999) 

Like data, information can be evaluated in quantitative and qualitative measures. 
Quantitative measures would include how many messages we sent out or how much e- 
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mail we downloaded. Qualitative measures would include whether those messages 
provided any useful advice on how we should do our jobs today, or helped us arrive at 
solutions for a particular problem. 

Davenport contends that data becomes information when its creator adds meaning 
or adds value in various ways, the most important of which all which begin with the letter 
C. They are: 

• Contextualized : Knowing for what purpose it was gathered. 

• Categorized : Establishing the units of analysis or key components of the data. 

• Calculated-. Analyzing the data mathematically or statistically. 

• Corrected : Removing errors from the data. 

• Condensed : Summarizing the data in a more concise form. 

A list of sports scores in a newspaper is an example of data. When the list is 
framed as one team defeating another, information results. As Peter Drucker said, 
“Information is data endowed with relevance and purpose.” (Drucker, 1992) Sports scores 
are only data, but when a reader sees the win or loss of a favorite team, he has 
information. 

3. Knowledge 

This, however, is still only information. To gain knowledge, one must know how 
to use this information - or in this case, understand how the game is played. Someone 
who did not understand the rules of golf, for example, might think that the player with a 
score of 83 did better than the player with a score of 72. But if he is able to understand 
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and internalize the information, and know that in golf the lowest score wins, then he can 
use this information to gain knowledge of the tournament being played. 

On a larger scale Davenport defines knowledge in this way: 

Knowledge is a fluid mix of framed experiences, values, contextual 
information, and expert insight that provides a framework for evaluating 
and incorporating new experiences and information. It originates and is 
applied in the minds of knowers. In organizations, it often becomes 
embedded not only in documents or repositories but also in organizational 
routines, processes, practices, and norms. (Davenport, 1999) 

Just what characterizes knowledge, then, is not so easy to identify. Following the 

chain we set up earlier, we can see that knowledge is derived from information, as 

information was derived from data. But where data consists of a computer printout, or 

information a newspaper sports page, knowledge resides in the minds of individual. 

C. ORGANIZATIONAL LEARNING 

1. Individual Learning 

In order to discuss organizational learning theories, we first must discuss the role 
of the individual in that learning. M. J. Marquardt points out that “individual learning is 
needed for organizational learning since individuals form the units of groups and 
organizations.” Also emphasizing the role of the individual is Peter Senge, who writes, 
“Organizations learn only through individuals who learn. Individual learning does not 
guarantee organizational learning, but without it no organizational learning occurs.” 
(Senge, 1990) 
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Placing somewhat less importance on the role of the individual in organizational 
learning is D. H. Kim, who argues, “Organizational learning is much more complex and 
dynamic than a mere magnification of individual learning. ... The level of complexity 
increases tremendously when we go from a single individual to a large collection of 
diverse individuals.” (Kim, 1993) 

Regardless of where these authors stand on the continuum of the importance of 
individual learning in organizational learning, they are in general agreement that 
individual learning is the starting point for organizational learning to take place. Without 
it, organizations as entities would be unable to leam on their own. 

2. Organizational Learning 

Organizational Learning is a concept that has gained widespread interest in both 
the business and academic communities. From the business side, the writings focus on 
paying attention to the concept of organizational learning in order to provide a sustainable 
advantage over corporate competitors. In the academic community, the writings take a 
procedural approach, looking at “detecting and correcting errors” or “encoding inferences 
from history into routines that guide behaviors.” (Argyris and Schon, 1978; Levitt and 
March, 1988) Though they come at it from slightly different perspectives, both groups 
would agree that organizational learning makes an organization more effective, profitable, 
and adaptable to change. Because of this, the organization should ensure processes are in 
place that allow individual and organizational alike to leam. 
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The military is not oblivious to this realization and, in fact, takes great pains to 
improve on detection and correction of errors. Still, organizational learning is the result of 
a wide range of factors and influences that include structure, environment, culture, and 
technology. The purpose of reviewing different organizational learning models is to 
identify those factors that may have significance to a military organization. As General 
Gordon R. Sullivan (U.S. Army) said: 

“Our task is not to make perfect plans...our task is to create organizations 
that are significantly flexible and versatile that they can take our imperfect 
plans and make them work in execution ” (Sullivan and Harper, 1996) 

3. Goals of NLLS Organizational Learning 

The goal of the Navy Lessons Learned System is an effort by the organization to 
promote Navy-wide learning through the collection and dissemination of all significant 
lessons learned, summary reports, and port visit reports, culled from the day-to-day 
maritime operations that are routinely undertaken. The Navy Lessons Learned System is 
designed to be the singular Navy Program for the collection, validation, and distribution 
of unit feedback as well as for the correction of problems identified from fleet operations. 
(OPNAVTNST 3500.37B) The goal of the system is to promote organizational learning by 
ensuring we leam lessons once. 

D. TYPES OF ORGANIZATIONAL LEARNING 

In this thesis I focus primarily on the Models of Organizational Learning 
presented by George Huber and by Chris Argyris and Donald Schon. Huber views 
organizational learning from a systems perspective primarily concerned with the 
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acquisition, distribution, and interpretation of information from the environment. Much of 
his writing emphasizes the need for the organization to provide conduits for the 
distribution of information. Argyris and Schon place greater emphasis on the individual. 
They contend that the way to change organizations is to change the way individuals 
respond to detecting and correcting errors. 

1. Huber’s Organizational Learning Model 

In his literature review on organizational learning, Huber describes the following 
four processes, or constructs, that contribute to organizational learning. 

a. Knowledge Acquisition 

Learning occurs when an organization acquires knowledge. This 
knowledge can be acquired intentionally by such means as organizational experiments, 
which result in focused learning, or unintentionally and unsystematically, which results in 
“haphazard or multi faceted” learning. (Huber, 1991) If the organization facilitates 
experimental learning, the focus should be on increasing the accuracy of feedback 
generated. Also, increased emphasis should be placed on determining the cause and effect 
relationships between actions and outcomes. 

Other methods of knowledge acquisition include “scanning the 
environment” for knowledge or facts, using information systems to store, administer, and 
retrieve this information, and the grafting of new members who “possess knowledge not 
previously available within the organization.” (Huber, 1991) 
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b. Information Distribution 

According to Huber, Information Distribution is a determinant of both the 
“occurrence and breadth” of organizational learning. (Huber, 1991) This distribution is 
the process by which an organization shares information among its members, thereby 
promoting knowledge. The more widely information is distributed in the organization, the 
more varied sources for it to continue to exist, thus making retrieval efforts more likely to 
be successful. Brown and Duguid concur with this, stating that a majority of learning 
takes place in informal “communities of practice” by members sharing stories or 
anecdotes of work. The greater the sharing or distribution of information within the 
entity, the greater the organizational learning that takes place. (Brown and Duguid, 1991) 

c. Information Interpretation 

Although information has been distributed throughout the organization, 
the intended receivers still must interpret it. Factors affecting the interpretation of 
information include the receiver’s cognitive map, or frame of reference. Researchers have 
established that cognitive maps can vary greatly across units within the organization. 
(Walker, 1985; Kennedy, 1983 et al) Because belief structures are automatically applied 
to any incoming information, understanding this interaction between cognitive maps and 
message interpretation is important to understanding how organizations learn. 

To aid in this information interpretation, the greater the media richness, 
the greater the possibility that the meaning intended by the sender will be realized by the 
receiver (Huber, 1991). Still, the distribution and media richness must be tempered, 


27 



because interpretation across organizational units is diminished if the amount of 
information exceeds the units’ capacity to process it. Information overload will likely 
result, with filters being applied and little or no information being processed. 

d. Organizational Memory 

Organizational Memory refers to the way an organization stores its 
information. It can be hard information, which is stored in standard operating procedures, 
routines, and scripts, or it can be soft information, which is corporate knowledge, 
experiences, or practices. For an organization to learn, it is necessaiy that its members be 
able to access the information stored or possessed by other members. The greater the 
specialization or departmentalization of an organization, the greater the chance members 
will not know what other members know. In order for an organization to demonstrate 
learning, learned behavior must first be stored in memory and then brought forth from 
memory. Having the information tucked away and inaccessible will discourage or defeat 
learning from taking place. (Huber, 1991) 

2. Argyris and Schon’s Organizational Learning Model 

In their book, Organizational Learning: A Theory> of Action Perspective, Chris 
Argyns and Donald Schon make a connection between the learning of the individual and 
the learning of the organization. They say the way to change organizations is primarily 
through changing individual actions. This book introduced the concepts of single, double, 
and deutero learning, to describe the way organizations learn by the detection and 
correction of errors. 
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a. Single Loop Learning 

As each member is assimilated into an organization he or she is 
indoctrinated through a process of socialization to a rule-governed behavior appropriate 
to the organization. As members carry out the practices common to the organization, they 
manifest a kind of knowledge. (Argyris and Schon, 1978) This reveals itself in each 
member as his or her own image, or map, of the organization. Taken as a whole, this 
results in organizational maps that are the shared description of the organization 
individuals have constructed. This shared description, or map, can include standard 
operating procedures, work flow diagrams, and any “actual patterns of activity” that are 
“guides to future action.” (Argyris and Schon, 1978) 

At times there are inputs to individual members that run counter to 
organizational norms. When there is a mismatch of expectation (detection of error), 
members seek to align activities to bring expectations back into norm (correction of 
errors). This is an example of single loop learning: 

Members of the organization respond to changes in the internal and 
external environments of the organization by detecting errors that they 
then correct so as to maintain the central features of the organization. 

(Argyris and Schon, 1978) 

Single loop organizational learning occurs when errors are detected and 
corrected, but the underlying norms and polices of the organization remain unchanged. 
Argyris and Schon’s example is that of a thermostat. When the temperature drops below a 
certain set value, the heater turns on. The thermostat makes no attempt to discover why 
the temperature has fallen below the prescribed norm. 
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b. Double Loop Learning 

Single loop learning works well when the organization is concerned 
primarily with achieving existing goals, but there are many cases when error correction 
requires that organizational norms themselves be modified. (Argyris and Schon, 1978) 
This higher learning requires an organization to understand its environment in order to 
increase the range of options available. For members to simply keep on doing what they 
already know, but better, is not enough to correct the error. In double loop learning, in 
addition to the detection and correction of errors, the organization is actively involved in 
the “significant restructuring and configuration of corporate norms.” (Argyris and Schon, 
1978) 


Double loop learning is those sorts of organizational inquiries that 
resolve incompatible organizational norms by setting new priorities and 
weighting of norms, or by restructuring the norms themselves together 
with associated strategies and assumptions. (Argyris and Schon, 1978) 

Double loop learning would occur if the thermostat could determine that 
the temperature dropped because the window is open, and then close it. 


c. Deutero-Learning 

It has become apparent to many organizations that even changing the 
underlying norms when a mismatch occurs is not enough. In order for a company to be 
continually innovative, it needs to learn how to carry out single and double loop learning. 
This type of “learning how to leam” is called deutero-leaming or second-order learning. 
(Argyris and Schon, 1978) 
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When a company engages in deutero-leaming, it discovers what occurred 
to facilitate or inhibit learning and invents new strategies for learning. (Argyris and 
Schon, 1978) 

In single loop learning, individuals are concerned with the effectiveness of 
existing norms and procedures without questioning underlying procedures. In double loop 
learning, they follow error detection by questioning the organizational norms themselves 
for error correction. If the organization becomes concerned with discovering how it 
learns, then it is engaged in deutero-leaming. 

E. INFLUENCES ON ORGANIZATIONAL LEARNING 

The previous sections highlighted two views on organizational learning. Though 
slightly different in their focus, they addressed the topic from a perspective that the 
learning of an organization was based either in part or in whole on the knowledge or 
actions of the individuals as an aggregate. While the individual is certainly in the 
forefront, Marlene Fiol and Maijorie Lyles inform us there are other factors, such as the 
structure, environment, and culture, that influence the probability that learning takes 
place. 

1. Structure 

According to Fiol and Lyles, a centralized mechanistic structure tends to reinforce 
past behaviors. A mechanistic organization is thus only capable of Argyris and Schon’s 
single loop learning. However, an organic, more decentralized structure promotes the 
shifts of beliefs and actions that allow for double loop learning. (Fiol and Lyles, 1985) To 
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encourage learning, management must therefore move away from mechanistic structures 
that encourage conformance to existing norms toward the organic, flexible structures that 
encourage reflective action-taking. (Fiol and Lyles, 1985) 

2. Environment 

While Fiol and Lyles acknowledge there is some disagreement over the different 
viewpoints on Organizational Learning, they point out that there is much more consensus 
on the importance of the environment to the organization. The “ultimate aim of the 
organization” is long-term survival, and to accomplish this “companies align with their 
environment” to remain competitive and innovative. This alignment implies that the firm 
must have the potential to learn, unlearn, or relearn based on its past behaviors. (Fiol and 
Lyles, 1985) 

This learning is conditional, however. If the internal or external environment is 
too complex for the organization to handle, no learning will take place. In order to leam, 
there must be both “change and stability.” Too much stability causes the organization to 
be static or dysfunctional. Too much change and the organization is unable to map the 
environment. Learning requires a balance between uncertainty and constancy. (Fiol and 
Lyles, 1985) 

3. Culture 

Culture can be defined as an organization’s values, beliefs, and norms, manifested 
as symbols, rituals, and language. Thus, culture will influence organizational action¬ 
taking. (Fiol and Lyles, 1985) It is these norms, or established patterns of behavior, that 
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can encourage or inhibit learning and partially determine strategy and the direction of 
organizational change. Changing the capacity of an organization to learn thus often 
involves the changing or restructuring of the norms and belief systems. 
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IV. USE OF TECHNOLOGY IN A LEARNING ORGANIZATION 

A. INTRODUCTION 

Each day we are presented with the latest and greatest advances in information 
technology, and all are being publicized as critical to an organization. Brand new 
technologies such as data mining, intranets, video conferencing, and distance learning are 
being pushed as solutions for the challenges we face as we leave the Information Age and 
enter the Knowledge Age. 

Today, the Internet presents us with almost limitless capabilities. E-mail, 
discussion groups, and database searching are just a few uses of the technology. The 
ability to reach almost anyone, at any time, presents us with an extraordinary opportunity 
to expand the organization’s learning potential. 

Still, not all technical experts and academic scholars are in agreement on the 
benefits realized from the increasing amounts of investment in Information Technology. 
Karl Sveiby contends that the confusion between knowledge and information has caused 
managers to sink billions of dollars in information technology ventures that have yielded 
marginal results. (Sveiby, 1997) Erik Bmjolfsson, a professor of Information Systems at 
MIT Sloan School, notes in Information Week: “The same dollar spent on the same 
system may give competitive advantage to one company but only expensive paperweights 
to another.” (Sept 9, 1996) 


35 




In this chapter we will look at the emerging issue: How can organizations use 
Information Technology to support a Learning Organization and improve organizational 
performance? 

B. BUILDING A LEARNING ORGANIZATION 

The 1990s have been characterized by a growing interest in the learning process. 
This has been caused by a widespread belief that learning and innovation are essential for 
an organization to survive. As a consequence of a rapidly changing environment, 
organizations are “going out of business everyday because they have failed to adapt to 
change or they have adapted too slowly.” (Ackoff, 1981) R. L. Ackoff, among others, 
believes that businesses that fail to become learning organizations will not survive. 
Echoing these thoughts is Peter Senge, who remarks, “The rate at which organizations 
leam may become the only sustainable source of competitive advantage.” (Senge, 1990) 

Though the military does not have to worry about going out of business, the 
lessons of Ackoff and Senge are more relevant than one would first think. There is a 
common axiom that the militaries of the world adopt doctrine to re-fight and win the last 
war. For example, the French built the Maginot Line to stop a German offensive based on 
correcting deficiencies of trench warfare identified in the latter days of WWI. Iraq built 
vast static defensive positions to counter mechanized coalition forces in Operation Desert 
Storm as a result of earlier success against waves of dismounted Iranian Infantry. Also, 
the United States Army’s attempts to overthrow Adid in Somalia were based on tactics 
developed in Vietnam to fight a guerrilla force, not on the Clan relationship that actually 
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existed. There is no greater rapidly changing environment than war, and the price of 
failure to adapt can mean the loss of life. Taking Ackoff s words to heart, we will look at 
what comprises a Learning Organization and how we can introduce organizationional 
learning into our organization. 

Before we move onto the process of building a learning organization, we must 
first define what a learning organization is and how it differs from organizational 
learning. David Garvin, noting that although “surprisingly, a clear definition of a learning 
organization has proved elusive over the years,” gives us perhaps the most 
comprehensible definition: 

A learning organization is an organization skilled at creating, acquiring, 

and transferring knowledge, and modifying its behavior to reflect new 

knowledge and insights. (Garvin, 1993) 

It is worth noting that some of the published literature uses the terms Learning 
Organization and Organizational Learning interchangeably. For our purposes, we will 
follow Michael Marquardt’s argument that there is a distinct difference between the two 
and that it can best be explained as a “process” vs. “product” argument. The learning 
organization is the desired end product. Organizational Learning is the cognitive process 
required to raise the capacity of the total organization towards that result. (Marquardt, 
1996) 

1. Five Building Blocks Of A Learning Organization 

Garvin believes that a learning organization has members experienced at five 
main activities. They are: systematic problem solving, experimentation with new 
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approaches, learning from their own experience and past history, learning from 
experiences and best practices of others, and transferring knowledge quickly and 
efficiently throughout the organization. (Garvin, 1993) We will look at these individually 
to see if and how they relate to the Navy and the Navy Lessons Learned System. 

a. Systemic Problem Solving 

Systemic problem solving entails an organization’s use of scientific 
methods for diagnosing problems, rather than guesswork. By using simple statistical 
tools, such as histograms and Pareto charts, organization members will be using data 
instead of their own assumptions to make decisions. The idea behind this is to get 
organizations to continually ask themselves, “How do we know that is true?” A common 
phrase in the military is, “That’s close enough for government work.” But if learning is to 
take place, we cannot be satisfied until we are certain that we have looked for the best 
answer. (Garvin, 1993) 

b. Experimentation 

Experimentation is described as “the systematic searching for and testing 
of new knowledge.” (Garvin, 1993) Testing new knowledge differs from problem solving 
in the fact that it is not seeking to find solutions for current problems. Instead, it is 
motivated by the desire to expand the horizon. Experimentation for Garvin takes two 
main forms: 

• Ongoing Programs that are a continuing series of small experiments 
designed to produce incremental gains in knowledge. 
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• Demonstration Projects that are more complex and involve holistic 
system changes introduced at a single site that have the goal of 
developing new organizational capabilities. 

Experimentation in the Navy is an active and ongoing concept. A major 
source of this takes place at the Maritime Battle Center at the Naval Warfare 
Development Command in the form of Fleet Battle Experiments (FBE). A FBE is 
designed as an experiment,, not an exercise. Its goal is to provide new doctrine or new 
insights into emerging technology in an operational environment. This will in turn 
generate ideas for further warfare concepts, or subsequent FBEs. (NWDC Public Affairs, 
2000 ) 


c. Learning From Past Experience 

Companies must review their past successes and failures and 
systematically assess them. Lessons must be recorded in an easily available form so 
employees have access to them. (Garvin, 1993) 

Garvin cites the example of Boeing’s work to overcome the design and 
production problems it experienced with the 737 and 747 by going back and comparing 
that process with the development process of the 707 and 727, two of the companies most 
profitable planes. A three-year study resulted in a one-inch thick book of lessons learned 
and recommendations. Several members of this team were then transferred to the 767 
start up, where they produced the most “successful, error free launch in Boeing’s history.” 
(Garvin, 1993) 
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British Petroleum takes the learning process a step further. The company 
has established a post-project appraisal unit that reviews major projects, writes up case 
studies, and derives lessons that are then incorporated into the organization’s planning 
guidance. The bulk of this unit’s time is spent in the field interviewing managers for their 
analysis. (Garvin, 1993) 

This part of Garvin’s Learning Organization most closely resembles the 
Navy Lessons Learned System, with a couple of exceptions. Unlike the Boeing example, 
the Navy Lessons Learned System does not currently do a trend analysis on the data. 
Second, where BP has an active collection process, the Navy relies on a passive input 
system for collection. 

d. Learning From Others 

Learning from others entails looking outside one’s own immediate 
environment. The idea is that the observations of other companies can at times serve as a 
catalyst for new thinking within your own organization. The commonly accepted term is 
“benchmarking,” an “ongoing investigation and learning experience that ensures that the 
best industry practices are uncovered, analyzed, adopted, and implemented.” (Garvin, 
1993) Still another way of securing an outside opinion is getting feedback from your 
customers or observing them actively using your product. 

While the relevance of these ideas to the military may be seem debatable, 
the idea behind them is not. In order to leant, an organization has to be receptive to new 
ideas. By being less defensive when criticism is offered, we allow new ideas to surface. 
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e. Transferring Knowledge 

In order to leam, the sharing of ideas and knowledge must be done quickly 
and efficiently throughout the organization. (Garvin, 1993) As a means of accomplishing 
this, Garvin lists a number of available methods, each with its own strength and 
weakness. 

Perhaps the easiest and most prevalent means of sharing is through the use 
of reports or tours. Garvin believes, however, that despite their popularity, reports and 
tours are a cumbersome way of transferring knowledge. Trying to absorb facts from a 
written report or from seeing a complex concept demonstrated is fairly difficult. It is 
much easier and more effective to actively experience it. From this observation, Garvin 
notes that personnel rotation programs are one of the most powerful methods of 
transferring knowledge. (Garvin, 1993) When expertise is held locally, those people who 
are fortunate enough to have daily contact benefit tremendously. By transferring the 
expert to different parts of the organization, companies are able to spread the knowledge 
around. 

Another extremely effective method of transferring knowledge is through 
the use of education and training programs that focus on problem solving. To be 
effective, the organization must follow through and ensure that this training is tied 
directly to implementation. (Garvin, 1993) In addition, the use of problem solving 
techniques can be increased if employees feel that their ideas will be listened to. One way 
of doing this is through the establishment of an awards system. When employees know 
that their ideas are being evaluated and that they have buy-in from the top, the 
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organization will have greater success in implementing new ideas and changing the 
entrenched status quo. 

C. MEASURING LEARNING 

Attempts to measure learning can trace their roots back to the early part of the 20 th 
century when businesses noticed that production costs typically fell with a corresponding 
increase in cumulative volume. This soon became known as the learning curve. Learning 
curves in the 80 to 85 percent range of their projected level were not uncommon. (Garvin, 
1993) 

For organizations trying to measure learning, however, these curves are 
incomplete. They focus on a single output, such as price, and do not address other values, 
such as quality or new product introductions. (Garvin, 1993) In order to measure 
learning, a new framework had to be developed. 

To track this progress, Garvin believes that organizational learning usually 
proceeds through three overlapping and measurable stages: 

• Cognitive Stage: Members of the organization are exposed to new ideas and 
begin to think differently. 

• Behavioral Stage: Employees begin to internalize new insights and alter their 
behavior. 

• Performance Improvement Stage: Changes in behavior lead to measurable 
improvements in results. 
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At the cognitive level, surveys and questionnaires are used to test the depth of 
understanding and to see if some of the meanings or terms are still unclear. (Garvin, 
1993) 

To evaluate changes in behavior, in addition to the surveys, direct observation 
must be made of the employees in action. This can been done by implementing quality 
control recordings of phone calls, placing orders to customer service desks, or by use of 
“mystery shoppers” to sample the product at particular stores. 

Finally, a comprehensive learning audit is undertaken to measure and assess the 
cognitive, behavioral, and performance changes to ensure the efforts have produced 
results. (Garvin, 1993) 

D. STEPS TOWARDS BECOMING A LEARNING ORGANIZATION 

Transforming an organization into a learning organization is not an easy 
undertaking. It requires a commitment from everyone involved, and the results may be 
slow in presenting themselves. Still, there are a number of concepts the organization can 
embrace immediately to expedite the process. 

The first step is to foster an environment that is conducive to learning. The 
organization should allow time for reflection and analysis. When members are not 
“hamed or rushed,” they will have time to identify cause and effect relationships, or 
assess current work systems, and learning will occur. (Garvin, 1993) 

The creation of learning forums, which are programs or events designed with 
explicit goals in mind, is another step. (Garvin, 1993) Asking provocative questions. 
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talking to customers, suppliers, or outside experts can open up the organization to new 
ideas. Turning to non-traditional sources for information or seeking opinions from people 
one would not normally consult can help generate new ideas. Division officers can seek 
input from members who work for them, asking them to come up with a different way of 
doing business. Anyone who hears the phrase “but that’s the way it’s always been done” 
should immediately question its validity. By employing a critical systematic thinking 
process, members of the organization can identify previously held assumptions and 
provoke feedback loops, which will provide and encourage learning. 

The last and most important step is to open up information flows. “Boundaries 
inhibit the flow of information,” and knowledge gained during routine operations will 
become compartmentalized, thereby reinforcing preconceived conceptions or habits. 
(Garvin, 1993) Opening flows can be done in a number of ways. Regularly scheduled 
conferences, meetings, or teams that are designed to cross levels in the organization are 
just a few. The greatest tools to cross platforms and connect people are the rapidly 
increasing capabilities of Information Technology, which can be used to generate, 
capture, and share knowledge throughout the organization. 

E. SUMMARY 

This chapter provides a frame of reference in which to understand the processes 
that make up organizational learning and some general concepts and definitions that will 
be used in later analysis. The importance of the individual as a basis for organizational 
learning was explored and discussed, as were several methods of acquiring knowledge. 
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Equal importance was given to external factors that are critical in understanding how the 
organization learns. Structure, environment, and culture all affect the individual’s frame 
of reference and the range of potential behaviors that may result. 
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V. INFORMATION TECHNOLOGY IN OTHER ORGANIZATIONS: THE 
CENTER FOR ARMY LESSONS LEARNED AND HAITI, A CASE 

STUDY 


A. INTRODUCTION 

It has been shown that organizational learning occurs when the knowledge and 
experience that reside in the organization’s members are shared throughout the 
establishment. In order for learning to occur in this way, however, the organization relies 
on its people and groups to be agents for this transfer of knowledge. As the organization 
matures, this knowledge is built into the culture, structure, and memory of the 
organization so that even though people leave the organization, the knowledge remains. 

Organizations do not have to rely solely on their members’ interactions with one 
another for sharing to occur. They can enhance this transfer of knowledge through the use 
of Information Technologies. 

IT’s increasingly widespread usage can contribute to the growth of organizational 
knowledge in two ways. First, it furnishes an extremely efficient way to collect the 
information that resides in its members, even those who reside in its farthest node. 
Second, it provides the members with an easily accessible means of retrieval of this 
information through the use of Internet and intranets, distributed databases, or knowledge 
portals. Storing the organization’s knowledge base using IT systems gives the added 
benefit of rapid modification of information as needed. For comparison to the Navy 
Lessons Learned System, we will look at another organization, the Center for Army 
Lessons Learned (CALL). Particular emphasis will be placed on CALL’S efforts to 
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implement Information Technology in order to become a more effective learning 
organization. 

B. HISTORY 

Like the Navy, in the mid 1980s the Army leadership realized that despite the 
investment in the National Training Center (NTC), there was no method to capture the 
lessons learned from training centers in the Mojave Desert. (CALL, 97-13) In order to 
accomplish this, in 1985 the Army created the Center for Army Lessons Learned (CALL). 
CALL’S initial focus was on the continental United States (CONUS) units as they rotated 
through NTC’s desert training. Today CALL operates collections centers at the Joint 
Readiness Training Center (JRTC) for light forces at Fort Polk, LA, the Combat 
Maneuver Training Center (CMTC) at Hohenfels, GE, and the Battle Command Training 
Program (BCTP) at Fort Leavenworth, KS. 


C. PROCESS 

The CALL staffs are continually looking out to identify events that are indicative 
of new or future missions in the Army. These events will then be candidates for a 
Combined Arms Assessment Team (CAAT), who will be used to gather data for lessons 
learned and organizational knowledge gained. 

Of particular recent interest to CALL were the contingency peacekeeping 
operations that have sprung up. Haiti was a prime example. By sending a team of experts 
to observe firsthand the new situations, CALL hoped to document and develop problem- 
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solving strategies on the spot. (Henderson, 1998) Of particular note, historically CALL’S 
approach was to widely deploy observation teams with little effort to define learning 
objectives. “This strategy resulted in the collection of massive amounts of raw data that 
overloaded the Army’s capacity to turn it into useful information.” (Henderson, 1998) 
Today, CALL selects events for observation that it feels has a “high potential” for 
providing data with significant value. 

Once an event has been selected, and prior to deployment, each CAAT team 
member will be trained as a “directed telescope” and provided a detailed set of questions 
about events they are to observe in the field. (Henderson, 1998) This helps to keep team 
members focused and provides a structure for the events they are observing. Collection 
personnel deploy with sufficient equipment and supplies that allow them to electronically 
pass observations and data back to CALL using digital equipment. This also allows team 
members to log on to the CALL database while in the field to access previous lessons 
learned information. (CALL, 97-13). 
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D. COLLECTION TEAM ORGANIZATION 



From Ref. [CALL, 97-13] 

CALL collection teams are composed of eight to twelve expert personnel who are 
tasked from various units in the Army. The majority of personnel are pulled from the 
schoolhouses (TRADOCs) and are selected for a particular skill that is relevant to the 
knowledge being sought. (Henderson, 1998) The person in charge of the team will 
usually be a high ranking officer (0-5 or 0-6) who is also tasked from the FORSCOM 
Army. To assist the collection team, CALL will provide three people, an Operations 
Officer (OPSO) to share responsibilities with the team leader, an Operations Non 
Commissioned Officer (OPSNCO), and a civilian analyst who is assigned to the team but 
remains at Leavenworth. 
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E. COLLECTION PROCESS 


Flow of Observations: 
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From Ref. [CALL, 97-13] 

Once the team is on station, the members join an army unit on patrol to observe 
and document events in real time. It is important to note that CALL observers are not 
evaluators. The CALL personnel are attached to the unit and become an integral part of 
the operation. (CALL, 97-13) They are there to support the unit as well as observe. 

For collection purposes, the observed information must be operational or tactical 
in nature. Once a problem is identified, the CALL team follows it back to its source, 
looking to identify systematic problems, not temporary anomalies. (Henderson, 1998) 
After a patrol is finished, the observations will be verified with the mission commander 
and discussed further with the other observation teams members. The observations are 
then sent to an analyst at Fort Leavenworth for early feedback. The analyst will discuss it 
with other experts and send back either more questions or possible solutions. 
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F. 


INFORMATION TECHNOLOGY TO SUPPORT LESSONS LEARNED 


1. Contingency Operations 

CALL and CAAT teams effectively utilize IT for integrating new knowledge. 
Within days of arriving in Haiti, the CAAT teams were sending raw data back to the 
analyst via satellite at the rate of five to ten observations a day. (Henderson, 1998) These 
unprocessed observations were posted on various electronic bulletin boards and also 
electronically distributed to a network of appropriate specialists for feedback. This 
allowed better questions to be devised, and after discussion, reflection, and feedback, the 
new information was edited for inclusion into the electronic organizational memory, or 
what we know as the lessons learned database. (Henderson, 1998) 

To supplement the observations, videotapes and descriptions of what happened 
were sent via regular mail. With these were diagrams of the events and what the 
observers noted were possible causes and consequences. 

The team returned to Fort Leavenworth after two months of patrols in Haiti. A 
three-day meeting then took place, during which they went through the observations, 
deciding what lessons to publish in an “Initial Impressions” report. This report was then 
sent out to thousands of ground and replacement troops. (Henderson, 1998) 

The team also produced over 100 vignettes of real world scenarios for 
replacement troops. These vignettes, along with videos of actual missions and the lessons 
learned information gained, were used to provide realistic training for replacement troops 
that was complete “right down to the barking dogs, rotting garbage, and belligerent 
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crowds.” The presentation so closely approximated conditions the soldiers would 
encounter that CALL was credited with providing for a seamless troop transition as the 
10 th Mountain Division was replaced by the 25 th Infantry Division troops. (Henderson, 
1998) 


2. Combat Training Center Operations 

Contingency Operations are not the only scenarios that CALL monitors for 
lessons learned. Ongoing training is conducted practically every day. This enables Corps 
to Squad level Army units to fight a well-equipped and well-trained enemy in a variety of 
terrains that range from barren desert to densely vegetated woodlands. (CALL, 97-13) 
This training takes place at the Combat Training Centers (CTCs). It is at these CTCs that 
the most continuous sources of observations and lessons learned are generated, and under 
the most realistic of situations. CALL uses these exercises and lessons learned to provide 
three essential products to the rest of the Army: the CTC Quarterly Bulletin, the CTC 
Trends and the Topic Newsletter. 

a. CTC Quarterly Bulletin 

The CTC Quarterly Bulletin publishes articles on techniques and 
procedures that work. It is authored by current or former CTC Observers, and its audience 
is primarily units that are scheduled for a CTC rotation. (CALL, 97-13) 
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b. CTC Trends 

(1) CTC Trends Bulletin. CALL receives trends and associated 
TTPs from all of the training centers on a regular basis. These are generated from unit 
rotations at the CTCs. The CTC branch of CALL organizes the trends and publishes a 
trends bulletin every six months. (CALL, 97-13) 

(2) CTC Priority Trends Compendium. All of the recurring trends 
and associated TTPs for each CTC are compiled into a compendium of priority trends, 
which are published annually. The compendium also contains a matrix chart that shows 
the number of times per quarter that a particular trend was documented over the previous 
two or more years. (CALL, 97-13) 

(3) CTC Trends Analysis. The CTC trend analysis is a two-part 
product. In the first part, for each CTC, CALL publishes a separate analytical review of 
each of the priority trends that were listed in the CTC Priority Trends Compendium. The 
second part is a cross CTC analysis of all CTCs. (CALL, 97-13) 

c. Topic Newsletter 

The Topic Newsletter picks a specific subject or issue. It then highlights 
the areas that are potential problems and finally provides useful tactics, techniques, and 
procedures to be used throughout the force. 

G. INFORMATION SYSTEMS DIVISION 

The Information Systems Division of CALL is the unit responsible for the 
facilitation of data collection, processing of observations, and dissemination of lessons 
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learned via CALL products. The primary IT tool used for this formulation of collection 
plans, categorizing of observations, and supplying of trend analysis is the CALL 
Collection and Observation Management System (CALLCOMS). CALLCOMS supplies 
the Army with an extremely effective process to automate the collection and analysis of 
lessons learned. Additionally, it provides training support packages tailored to specific 
units and publishes numerous periodicals that directly address the correction of mistakes 
for unit commanders and the total Army. An historical record of past trend analysis of 
NTC units is available to anyone with a desktop computer via the CALL homepage. 
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VI. FINDINGS 


A. INTRODUCTION 

This section describes the research method and sources of archival data collected 
during research. An in-depth survey with a large population was beyond the scope of the 
thesis. Instead, an exploratory questionnaire was sent to a number of Naval Postgraduate 
School students who have recently returned from the fleet. This resulted in an extremely 
low number of usable results. 

A request was also made to a small number of fleet units. The resultant response 
here, too, proved insufficient for analysis. In an effort to estimate the exposure of NLLS 
to the fleet, a different approach was undertaken. To obtain as wide a range of experience 
as possible, while being cognizant of the extremely low return rate, individual 
communities were specifically chosen to provide input for the survey. This was 
accomplished by soliciting various members of the targeted warfare specialties until an 
affirmative response was attained from a member of that community. 

The questionnaire was divided into three sections. The first section covered the 
respondent’s background, community, and experience. The second section was aimed at 
the Navy Lessons Learned Submission Process. Here, the questions were concerned with 
the respondent’s knowledge of the NLLS and frequency of individual or command 
lessons learned submission. The final section covered the retrieval of prior lessons 
learned as well as the respondent’s sense of the relative effectiveness of the system. 
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It is important to note the survey was not intended to be suggestive of a Navy¬ 
wide representation. Instead, it was designed to be a generalized, sample-based 
questionnaire specifically for the purpose of this research. In light of the data, it is 
recommended that a larger sample size survey be undertaken. All questions were asked 
in the past tense to reflect the time the respondents were in the fleet rather than stationed 
at the Naval Postgraduate School. 

B. SAMPLE DEMOGRAPHICS 

The sample candidates were individually chosen based on a community that they 
represented. The pool of respondents consisted of two Aviators, two Surface Warfare 
Officers, one Intelligence officer, one Submariner, one Medical Services Officer, and one 
Fleet Support Officer. 

The respondent’s rank ranged from a junior lieutenant to a senior lieutenant 
commander. Two of the respondents had prior enlistment time. The positions held while 
on sea duty covered a wide range of operations experience (Squadron Ops Officer, 
Combat Information Center Officer, Assistant Navigator, and Sub Pilot) as well as shore 
based IT related positions (ADP Officer and IT Project Officer). All but one had served 
aboard a ship, and those who had served aboard ship had made at least one deployment. 
Half of the replies came from personnel that had served onboard large decks (CVN or 
LPD). The average number of deployments of those who did serve aboard ship was two. 
The sample was composed of seven male and one female Naval Officers. 
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c. 


RESULTS AND DISCUSSION 


1. The Navy Lessons Learned Submission Process 

a. Have you ever heard of the Navy Lessons Learned Center? 

This first question was designed to get an overall feel for the respondent’s 
knowledge of the NLLS in order to be able to more accurately draw conclusions from the 
rest of the survey. Of the eight, five, or 63 percent, had heard of NLLS. Two of the “no” 
answers came from aviators, and one from the submariner. 


yes no 


t 1iF 

3 


i 


El Yes 
■ No 


b. Did you know that the Navy Lessons Learned Center has a Web 
site with a database of all the Lessons Learned submitted in the 
past two years? 



Yes No 
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This second question was designed to assess the respondent’s knowledge 
of the NLLS system. Whereas most people had heard of the system, only 25 percent knew 
about the Web site. 


c. When your command concluded an exercise, were the lessons 
learned collected and forwarded to your ISIC? 


i 

Yes Sometimes No Unknown Local 

The majority of the respondents, or 75 percent, felt that lessons learned 
were being collected either all of the time or at least some of the time. There was one 
“no” answer, and that came from the Medical Services Officer. 

Breaking down the “yes” answers, three of the six collected lessons 
learned after every exercise, two of the six after most exercises, and one of the six 
collected lessons learned, but these were known to stay within the command (Fleet 
Support). The sole “unknown” response came from the Intel Officer. 


□ Yes 

■ Sometimes 

□ No 

□ Unknown 

■ Local 
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d. During a scheduled exercise or evolution, was lessons learned 
collection given consideration? 



The majority of respondents indicated that their commands gave lessons 
learned collection consideration during planning. The two negative responses were from 
the Intel Officer and the Medical Services Officer. 


e. Did you consider submitting a NLLS report a worthwhile task? 



Almost every respondent had a pessimistic reaction toward the utility of 
submitting lessons learned. The two positive responses came from the Submariner and 
the Fleet Support Officer. 
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2. The Navy Lessons Learned Retrieval Process 

The previous section revealed that almost everyone was either familiar with the 
lessons learned system or was a participant in submitting lessons learned. But 
organizational learning requires that knowledge gained by one member or unit be 
transferred to the other members. This section of the survey was aimed at determining if 
members were seeking out the knowledge that resided in nodes other than their own. 


a. Did your department have access to the SIPRNET? 



□ Yes 
■ No 


The most up-to-date lessons learned information can be accessed via the 
SIPRNET. In the event the service member does not have access to the SIPRNET, CD- 
ROMs with the same information are distributed on a quarterly basis. 

This question is getting at two important issues, the ability to access the 
most current observations, and a means for providing a conduit to receive audio, video, or 
bulletin board information that can be used to enhance the recipient’s understanding of 
the message. 

Two of the three negative responses came from the Submariner and the 
Medical Services Officer. For the Submariner, the lack of SIPRNET is understandable. 
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The MSO’s lack of SIPRNET can most likely be attributed to that community’s lack of 
secure need. The final negative response came from Aviator One. The ship he served on 
was non IT-21 compliant at the time. 

b. Have you visited the NLLS web or SIPRNET site? 


□Yes 
■ No 

Yes No 

Only one of the eight had actually visited the NLLS Web or SIPRNET 
site. Because this positive response came from the Medical Services Officer, it was 
unexpected. In answering the previous question the MSO cited a lack of SIPRNET 
access. When queried about this response, the respondent indicated he had accessed the 
NLLS Internet Web site. The NLLS Internet Web site provides an overview of the system 
but does not contain actual lessons learned. It does provide information on how to access 
them via the SIPRNET as well as provide a means for inclusion on the CD-ROM 
distribution list. 
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c. 


Did you have access to either the classified or unclassified 
Lessons Learned CD-ROM? 



□ Yes 
■ No 

□ Unclass 


Sixty-three percent responded that they did not have access to the NLLS 
CD-ROM. There were two affirmative answers from the Surface Warfare Officers, and 
the Medical Services Officer proffered the single unclassified answer. 

The high number of negative answers can most likely be attributed either 
to the way the question was asked or to a lack of respondents’ knowledge of the 
availability of the CD-ROM. It is certain most ships receive the lessons learned CD-ROM 
at a department level. If the problem is a lack of knowledge of the CDs, increasing 
awareness may help to resolve this. 
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d. Have you searched the Lessons Learned Data Base for a Lessons 

Learned? 


Yes No 

Most respondents answered “no” to this question. Of the three “yes” 
answers, two came from the Surface Warfare Officers and one from the Medical Services 
Officer. 

e. What type of Lessons Learned were you searching for (Previous 

exercise, similar exercise, port visit, etc.)? 


Port Prev IT 
visit Deply 

This question garnered a high level of affirmative responses. Five out of 
the eight respondents reported accessing the database or, more correctly, a database. The 
two Surface Warfare Officers were looking for port visit lessons learned. The two 
Aviators, who previously answered negatively to searching the NLLS database, explained 
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they had searched or read Air Wing Lessons Learned information. The last response was 
from the Medical Services Officer, and he wrote he was searching for IT lessons learned. 

f. On a scale of one to five, where one is of little use, three is of 
moderate use, and five is of exceptional use, rate the usefulness 
of prior lessons learned as they applied to you in a exercise or 
evolution. 


4- 

3.5 



-1 ■ . I I 1 1 1 I 11 

Aviatori Aviator2 SWOI SW02 Intel 


□ Aviatori 

■ Aviator2 

□ SWOI 

□ SW02 

■ Intel O 


The aviators rated the usefulness of lessons learned fairly high, with both 
giving lessons learned research prior to an exercise a four. Both of the SWOs stated that 
they used it for port visit information, but one gave it a three, and the other responded 
with a rating of one. The last respondent, the Intel Officer, gave it a moderate three. 

g. On an average of how many times a month did you access the 
lessons learned database? 

This question revealed that all participants very rarely accessed the 
database on a regular basis. One respondent (SWO) indicated that maybe in a good month 
he would access it twice, but for the remainder the answer was zero times or not 
applicable. 
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h. When your command prepared for an exercise, was the lessons 

learned database consulted during planning? 


Yes No 

This question resulted in two affirmative responses, one by a SWO and the 
second by the Submariner. However, in both cases, they clarified their responses and 
added that the lessons they consulted were on their own ship’s database. 

i. What type of circumstance would cause you to reference the 

lessons learned database? 

The modal response to this question was that a service member would 
access the lessons learned database if he or she was going to take part in or undertake an 
infrequent evolution. The Fleet Support Officer, Submarine Officer, and one of the 
Surface Warfare Officers expressed this feeling. Other common responses were that 
reference was made prior to an exercise, which was stated by the Intelligence and Medical 
Officers, and if the respondent believed there was corporate knowledge residing in the 
database, which was expressed by the two aviators. 
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3 . 


NLLS Effectiveness 


a. Did you consider reviewing the NLLS database prior to an 
exercise a worthwhile task? 

The two respondents who answered “yes” to question eight of the retrieval 
process both answered “yes” to this one. Of the other six “no” or “not applicable” 
responses, four felt if they had known it was there, they would have felt it a worthwhile 
task. The two remaining answers were from the aviators who remarked that reading 
lessons learned prior to an exercise was very much exercise dependent. Typically each 
exercise tended to be pretty specific. After the exercise started, lessons learned were 
almost always read, either from previous missions or those put out during a squadron or 
ready room meeting. 

b. Had referencing the Lessons Learned database resulted in a 
change in the way you would approach tactics, techniques, or 
procedures (TTP)? 


Yes No N/A 

The two “yes” answers in this question came from the aviators, though 
both stated that they did not refer to NLLSDB Lessons Learned; it was Air Wing or 
Squadron Lessons Learned. Both stated that any change that resulted was either in 
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techniques or procedures. Tactics generally were developed during Air Wing training, and 
the exercises sought to improve on them. 

Of the three “no” answers, two came from the Surface Warfare officers with no 
explanation, and one from the Submariner, who stated that if he had known it was there, 
it might have changed the way he would do something. The remaining three “not 
applicable” answers were from the Fleet Support, Intelligence, and Medical Services 
Officers. 


c. Where would you say you received the majority of your lessons 

learned information? 



H Professional Journals 
■ Message Traffic 
□ Meetings 


The responses received from this question can be grouped into three main 
themes: professional journals, message traffic, and meetings with people who shared 
corporate knowledge. Three out of the eight cited their community’s periodicals 
(Approach or Fathom), which they read on a regular basis. Two who cited message traffic 
said it included mishap reports. And the final two cited either scheduling or planning 
conferences or the informal network of fellow co-workers as the source. 
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d. Do you have any inputs that you feel would make NLLS a more 

effective tool? 

Four of the respondents felt NLLS would be a more effective tool if the 
awareness of the system could be improved. This conclusion can be anticipated in part 
because of their lack of knowledge of the system going into the survey. 

One reply from a Surface Warfare Officer sought to make it more user 
friendly. He felt that writing each lesson learned took a tremendous of time, since it had 
to be dissected or parsed into rigid NLLS format (observation, recommendation, etc). 

The remaining three provided no input to improve the product. 

D. ARCHIVAL DATA RESEARCH 

1. Introduction 

As part of this research, an analysis was conducted to determine the feedback 
generated from one of the most strenuous levels of training encountered during a 
battlegroup’s preparation for deployment, the Joint Task Force Exercise. (JTFEX) 

2. Background 

Following a deployment, ships usually enter into a planned maintenance and 
upgrade yard period. Upon completion of this yard period, the ship will once again rejoin 
the fleet returning to sea. These first few at-sea periods consist of individual or unit level 
training. Following this phase, the ship will undergo an extremely high intensity 
integrated, underway and inport interdployment workup cycle (DDTC) that culminates in a 
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certification exercise called the Joint Task Force Exercise. Throughout this exercise the 
Carrier Battle Groups (CVBG) and Amphibious Ready Groups (ARG) are presented with 
a scenario that emphasizes expeditionary warfare in a joint littoral environment. These 
exercises, which can involve many thousands of personnel, present increasingly complex 
scenarios to the participants. 

For example, during JTFEX 99-1, more than 24,000 U.S., Joint, and Allied 
service members were involved in exercises that took place from the coast of Virginia to 
the Islands of Puerto Rico. In addition to the Army, Navy, Air Force, and Marine forces, 
elements from Great Britain, the Netherlands, France, Belgium, Canada, Germany, and 
Bolivia participated. The JTFEX is designed to provide realistic training to fully prepare 
the ships and personnel for any and all operations they may encounter when deployed. 
The participants train using equipment and systems that represent the latest advances in 
technology. During these exercises forces are tested on their ability to deploy rapidly, 
conduct joint operations, and refine tactics, techniques, and procedures. 
(http://www.chinfo.navy.mil/) JTFEX’s have been called graduate level training for the 
battle groups, and successful completion is required prior to being certified ready to 
deploy. 

3. Methodology 

To provide input, a search of the Navy Lessons Learned Data Base was conducted 
for the last two years of observations. This resulted in returns for five Joint Task Force 
Exercises. The following three data points were investigated: 
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• Total number of lessons learned for the exercise. 


• Time required to validate the lesson learned. 

• Outcome resulting from the lesson learned submission. 

These outcomes were categorized into four classifications. The lesson learned 
submission could result in a Tactical Development Evaluation update (TACMEMO), 
Remedial Action Program (RAP) working group submission, “disagree” with the 
observation, or be delineated by either no remarks or “concur” with the writers’ 
observation. In both of these latter cases, the outcome was classified as “concur.” 

The significance of the four categories is marked by what further action the lesson 
learned generates. A TACMEMO update can result in a change to existing doctrine, 
tactics, techniques, or procedures currently in place. A RAP item identifies a deficiency in 
policy, organization, training, education, or equipment. In both of these cases the 
knowledge gained from the lesson learned will be promulgated throughout the 
organization in the form of new publications, doctrine, or equipment. 

In the case of a lesson learned classified as “concur” or no comments noted, for 
this knowledge to be disseminated, the organization’s members have to actively search 
the database. 


72 


4. 


Results 


a. JTFEX 98-1 



Lessons TACMEMO RAP Concur 
Learned 


The first JTFEX took place from January 12th to February 4th of 1998. 
JTFEX 98-1 consisted of the USS John C. Stennis Battlegroup (CVBG) and the USS 
Wasp Amphibious Readiness Group (ARG). The lessons learned for JTFEX 98-1 consist 
of five entries. Validation for these five entries took four months, and the outcome for all 
five is classified as “concur.” 


b. JTFEX 98-2 



'--1--—i . i "i 

Lessons TAC RAP Concur Disagree 


Learned 

* 
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JTFEX 98-2 was held from April 27th through May 13th. It consisted of 
the USS Dwight D. Eisenhower Battlegroup and the USS Saipan Amphibious Readiness 
Group. The lessons learned for JTFEX 98-2 consisted of 14 observations. Validation for 
these observations took five months and the outcome consisted of three TACMEMOS, 
ten “concur,” and one “disagree.” 


c. JTFEX 99-1 



Learned 


□ Lessons Learned 
■TAC 

□ RAP 

□ Concur 


JTFEX 99-1 took place from February 12th until March 7th. Taking part 
were the USS Theodore Roosevelt Battlegroup and the USS Kearsarge Amphibious 
Readiness Group. The lessons learned for JTFEX 99-1 consisted of 29 observations. 
Validation for these observations took four months and the outcome consisted of one 
TACMEMO, one RAP, and 27 “concur.” 


* 
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d. 


JTFEX99-2 



Learned 


□ Lessons Learned 
■TACMEMO 
ORAP 

□ Concur 
■ Disagree 


JTFEX 99-2 was held from July 18th through July 30th. Taking part in it 
were the USS John F. Kennedy Battlegroup and the USS Bataan Amphibious Readiness 
Group. The lessons learned for JTFEX 99-2 consisted of 11 observations. Validation for 
these observations took two months, and the outcome consisted of ten “concur” and one 
“disagree.” 


e. JTFEX 00-2 



Lessons TACMEMO RAP Concur 
Learned 


JTFEX 00-2 took place from May 10th until May 20th. The USS George 
Washington led the carrier battle group, and the USS Saipan led the Amphibious 
Readiness Group. The lessons learned for JTFEX 00-2 consisted of 27 observations. 
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Validation for these observations took seven months, and the outcome consisted of one 


TACMEMO, four RAP items, and 22 concurs. 


/. Overall 



Lesson TACMEMO RAP Disagree Concur 
Learned 


Overall for the five Joint Task Force Exercises, there were 86 Lessons 
Learned in the database. From these 86 observations, five TACMEMOSs and five RAP 
items resulted. Of the remaining 76 observations, two were classified as “disagree” and 
74 “concur.” 
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VII. ANALYSIS 


A. INTRODUCTION 

The goal of this section is to provide analysis of the Navy Lessons Learned 
System to determine if Organizational Learning is taking place. Using the theories of 
Huber, Argyns and Schon, and Fiol and Lyles as a basis for exploration, we examine 
whether there are systems in place that encourage and enable learning. In addition, 
emphasis is placed on examining the results of the sample survey to see if there are 
cultural or environmental factors whose effects could result in rendering even a well- 
designed information system ineffective. 

B. IS ORGANIZATIONAL LEARNING TAKING PLACE? 

1. Introduction 

This analysis uses a framework adapted from a Department of the Navy model 
that presents the requirements for a balanced knowledge management approach. 
(Navigating the World of Knowledge, 2000) This model states that balance is required to 
ensure that undue emphasis is not placed on any one single aspect of management. It 
contends that technology alone is not sufficient for knowledge transfer. The behavior of 
people must be changed, and they must be provided with the tools to use the technology. 
(Navigating the World of Knowledge, 2000) The framework comprises five separate but 
equally important pieces: Content, Process, Culture, Learning, and Technology. Using 
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this as a framework provides us with a useful platform from which to analyze the 
organizational learning models discussed earlier with the Navy Lessons Learned System. 



Figure 2. Organizational Learning Model 


2. Content 

Organizational Learning begins at Content. For the NLLS system this content 
manifests itself in the database of lessons learned. As we have seen, content is made up of 
much more than a flat file of information. For lessons learned to be credible, the content 
must be timely, relevant, and provide value for the user. 

For our purpose, timeliness of the information relates to how long it takes for the 
information or knowledge gained by one unit to be shared throughout the organization. 
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What is timely also can depend on the urgency of the situation or the significance of the 
lesson learned. If a ship is engaged in hostilities, any delay in sharing information may 
result in losses. If the ship is tied up to the pier, the applicability of the knowledge may 
not be apparent or needed for many months. 

For the research done on Joint Task Force Exercises, the Navy Lessons Learned 
System’s average time from submission to publication was a little under four and a half 
months. For organizations that do not have SIPRNET access and rely on the next CD- 
ROM distribution, an additional three months might be necessary. JTFEXs are held 
approximately every five months. In this instance, getting the information from the 
previous battlegroup experiences in time for preparations and planning may not occur. 

Relevance and value are closely related. Relevance can be thought of as the 
applicability of the information to the individual’s daily routine or task carried out. Value 
is the benefit the individual receives from accessing this information. The value of the 
information can diminish with the increased time required to obtain it. 

A frequent comment heard when discussing lessons learned has to do with the 
relevance or applicability of the information to the individual. The database comprises 
approximately 24,000 lessons learned. It is apparent from the research done on the 
JTFEXs that most lessons learned are very specific in nature, and the applicability for 
other individuals would appear to be very low. There are full text search engines available 
to look for certain topics in the database, but if the intention is to get an overall 
impression of the exercise prior to undertaking it, the information is generally too specific 
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to be of use. The reason for this becomes evident as one searches through the 
information. 

The JTFEX database comprises 86 observations, and the complete NLLS database 
is approximately 24,000 lessons learned. But most of these observations consist of a 
sample size of one, which presents individuals and analysts with difficulties in 
determining how valuable the knowledge might be. Most of these observations lack the 
richness or breadth needed to provide a wide increase in organizational knowledge. On 
the other hand, some events provide a wealth of information that is applicable 
organizational wide. A recent example is the bombing of the USS Cole and the lessons 
learned that could be gained from that ship’s experiences refueling in a foreign port. 

On a somewhat smaller scale are the lessons learned with every aircraft mishap or 
ship’s collision. In any of these instances trained investigators are dispatched to examine 
events that led up to the incident. From their analysis, significant lessons learned are 
written up and disseminated to the fleet. When the sample size of one consists of 
something as rich as an aircraft or ship mishap, the relevance to every aircrew or Officer 
of the Deck is apparent. The value is much less apparent when searching through a large 
numbers of specific instances for one that may be relevant. 

3. Technology and Process 

In Huber’s work the role of technology in organizational learning was explicitly 
specified as benefiting Organizational Memory. He stated that the hard information of a 
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company was resident in computer memories with the soft information and the ability to 
pass this on residing in the members. (Huber, 1991) 

With the far-reaching effects of the Internet, today’s technology can have a 
profound effect on not only Organizational Memory, as Huber theorized, but on the other 
three processes of learning, Knowledge Acquisition, Information Interpretation, and 
Information Distribution, thus dramatically increasing a member’s ability to pass this 
knowledge throughout the organization. 

a. Knowledge Acquisition 

Knowledge Acquisition is facilitated by NLLS when available 
technologies (Web based gateways) are used to scan the environment (NLLSDB) for 
information retrieval. For the military members using NLLS, this is a closed system, as 
the learning that occurs comes from within the organization. 

Knowledge Acquisition also can emerge from the revision of existing 
doctrine or innovations in strategy that result from inputs to the system. TACMEMO 
publications and RAP item submissions are the best examples of this. 

Lessons learned also have shown themselves to be generators of “Best 
Practices” that are considered to be superior in approach and results. Usually Best 
Practices represent SME experiences, but they also have been based on inputs from the 
fleet or research conducted in response to an observation. 
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b. Information Distribution 

Information Distribution is the process by which an organization shares its 
information among its members, thereby promoting knowledge. For the NLLS system 
this is accomplished in two parts. The first is the distribution by either the quarterly 
circulation of the CD-ROM database or the weekly updating of the SIPRNET database. 
The second is the accessing of the web site by members. 

The Navy Lessons Learned Organization information distribution process 
takes the Field of Dreams approach, the “If you build it, they will come” mindset. But 
relying solely on this approach is risky. Based on information provided by NWDC, 
during the period of May to October of 2000 the total number of lessons learned accessed, 
excluding port visits and RAP items, was 27,819. This breaks down to approximately 
4,600 hits a month. For the research on the Joint Task Force Exercises, I accessed 86 
lessons learned. In the fleet the average number of lessons learned searched is unknown 
but would probably be closer to twenty per person. Twenty lessons learned accessed per 
person, per visit, divided by the 4,600 hits, would mean that approximately 230 people a 
month visited the site. Since the majority of the information contained in the database is 
only provided if the user accesses the site, there is a minimal amount of Information 
Distribution occurring, even though the technology allows all members access. 

c. Information Interpretation 

Process in this framework is the means by which the organization 
captures, categorizes, and presents information. As important as capturing the 
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information is, capturing the context is even more so. Huber talks about this as 
Information Interpretation, or the manner by which distributed information can be given 
one or more commonly understood meanings. (Huber, 1991) Context is unique at any 
point in time, and a change in context can result in completely different interpretations 
between individuals. These interpretations may be subject to changing environment, 
recent events that have transpired, or even interactions between crew members on a ship. 
A knowledge base of lessons learned is particularly sensitive to the difference in context 
that exists between the sender of the message and the receiver. Using the Joint Task Force 
Exercises as an example, a particular lesson learned has behind it a host of conditions that 
may or may not be applicable or apparent to the receiver. Everything from different 
equipment configurations onboard ship to battlegroup composition challenges the 
relevancy and comprehension of the intended meaning to another individual. 

Compounding this difficulty, the lessons learned layout consists of a four- 
paragraph text only format. These paragraphs are broken down into an Observation, 
Discussion, Lesson Learned, and Comments section. To give the intended meaning a 
greater chance of being understood, Information Technology can be used as a way to 
enrich the message. Providing an opportunity to include drawings, diagrams, or 
documentation in a PDF format may make the information more understandable. Anyone 
who has tried to program a VCR understands the benefits of diagrams that are used to 
supplement the instructions. 

To increase the breadth of lesson learned applicability. Information 
Technology can also be used to cluster or bring together data and information that is 
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similar or related. Clustering - for example, on a keyword search - would allow the user 
the ability to compare and contrast information from a variety of sources around a subject 
or topic area. Even if the lesson learned is not applicable, or the search returns without a 
match, there may be valued gained. 

4. Learning 

a. Individual vs. Organizational 

To assess NLLS effectiveness on learning, individual learning must be 
separated from organizational learning. Individual learning involves the detection and 
correction of errors. It can consist of single loop or double loop learning. In single loop 
learning, members of the organization respond to environmental changes by detecting and 
correcting errors in ways that allow the organization to continue with its present policies. 
In double loop learning, the error is detected and corrected in ways that involve the 
modification of the organization’s underlying norms, policies, and objectives. (Argyris 
and Schon, 1978) Learning at an individual level, however, remains within the party or 
unit that created it. 

Organizational learning occurs when members of the organization share 
their memories, associations, or experiences. Sharing can be done by traditional methods, 
such as meetings and memorandums, or it can be done by computer-generated means 
such as e-mail, electronic bulletin board, or file downloads. This organizational learning 
is influenced by many factors, such as structure, strategy, environment, technology, and 
culture. (Fiol and Lyles, 1985) Organizational learning occurs with the acquisition and 
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transfer of knowledge, and it reveals itself when the organization modifies its behavior to 
reflect this new knowledge. (Garvin, 1993) 

Although learning remains essentially the same in the individual case as in 
the organizational case, the learning process is fundamentally different at the 
organizational level. (Kim, 1993) When we talk about organizational learning, we are 
concerned with conveying knowledge from one person to the next. 

b. NLLS Effect on Organizational Learning 
To evaluate NLLS effect on organizational learning, one has to look at the 
individual’s usage of the system. Usage in this instance is defined by an individual 
submitting a lesson learned to distribute newfound knowledge, or by an individual 
accessing the information to increase the breadth of the organization’s knowledge base. It 
is not possible to accurately evaluate the number of times that CD-ROMs are used to 
search the database. Instead, this opinion is based on the research data and information 
provided by the Web site. 

Overall, the sample survey and Web counter information indicate a very 
low rate of accessing the NLLS database by military members. Looking at inputs to the 
system, it’s important to remember that the database contains only those lessons learned 
forwarded by the individual’s Immediate Superior In Command (ISIC), not the total 
number of actual submissions that may have been received after an exercise. That said, 
for the five Joint Task Force exercises, which can involve up to 20,000 participants each, 
the average number of lessons learned generated is 17 per exercise. 
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For organizational learning to occur through the NLLS system, active 
participation by individual members is required. The NLLS system relies on a passive 
collection method and an active distribution method. Inputs are generated from 
individuals in the fleet. Once these observations are validated, CD-ROMs are distributed 
and the NLLSDB is updated. To complete the dissemination of knowledge, members 
must actively pull the information from the database. 

Currently, the Navy Lessons Learned System provides an extremely 
efficient way to promote organizational learning with the automation of submission and 
validation of lessons learned. However, the effectiveness of organizational learning 
suffers from the poor response rate or participation rate by members of all communities. 
To demonstrate organizational learning, “that which is stored in organizational memory 
must be brought forth from memory”. (Huber, 1991) This poor response rate inhibits the 
communication of new approaches or success stories gained from within the organization. 
Without greater involvement, formalizing the process of transferring best practices and 
lessons learned is not going to result in new discoveries or solutions being leveraged off 
existing ones. 


5. Culture 

Lessons learned collection has become synonymous with exercise completion. All 
too often it is given perfunctory thought, and the only reason inputs are generated is that 
they are mandated for after-action reports or required from the chain of command. More 
significant, even this obligatory emphasis placed on lessons learned collection is rarely 
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placed on lessons learned research prior to an exercise. The reasons can vary, but the 
research is often seen as an inefficient use of time by personnel undertaking an 
assignment or planning an operation. Frequent comments echoing this came from 
members who stated that they experienced a lack of benefit from the effort expended. For 
the NLLS system the greatest hindrance was in the lack of exposure most people had to 
the database. Still, it is not certain that with greater exposure the response rate would 
have fared much better. The small number of lessons learned generated from the five 
Joint Task Force Exercises indicates there is a fundamental culture present in the Navy 
that reduces lessons learned submission and hampers NLLS ability to promote 
Organizational Learning. This culture may be based in part on the mindset seen in 
Knowing Organizations, an attitude that hinders the learning organization’s culture of 
“openness to experience, encouragement of risk taking, and willingness to acknowledge 
failure.” (McGill and Slocum, 1994) 

The Knowing Organization is a model advanced by M. E. McGill and J. W. 
Slocum that can be seen in operation in virtually every franchise operation in the United 
States today. In a Knowing Organization, there is a belief that there is only one best way 
to complete a task, manage an employee, or structure the organization. 

Undoubtedly the most famous and visible of all knowing organizations is 
McDonald’s. Its 13,000 plus stores exemplify the best of what a knowing 
organization has to offer: efficiency, predictability, and control in 
production and customer services." (McGill and Slocum, 1994) 
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The best way to fry a hamburger, or how long the french fries should be cooked in 
oil, has been discovered and is committed to the organizational memory. A Knowing 
Organization focuses on “standardized policies, procedures, rules, and regulations.” 
(McGill and Slocum, 1994; Hannah, 1993) Knowing Organizations take their strength 
from the high levels of conformity and standard operating procedures that form a very 
effective organization as long as the environment remains relatively stable. 

In times of change, the Knowing Organization may make “incremental changes to 
existing processes but these are really only tweakings and fine tunings of what the 
organization already does.” (McGill and Slocum, 1993; Hannah, 1993) In a Knowing 
Organization, managers develop practices over time that protect the organization from 
pressures and outside influences, whatever their merit. “Rituals, stories, jargon, and 
physical settings are all instruments that can hinder their ability to detect and develop the 
mechanisms that learning requires.” It is because of this that McGill and Slocum believe 
Knowing Organizations are “learning disadvantaged.” (McGill and Slocum, 1993; 
Hannah, 1993) 

Lessons learned are often not written because we use the established doctrine and 
standard procedures. Doctrine says how to assault a beach, so we plan and execute 
according to doctrine. If a lesson learned is written, it tends to deal with communication 
issues, circuit management, or even the benefits of e-mail, something that may have great 
applicability to the ship involved at the time, but marginal utility to the next battlegroup. 

Small incremental changes to existing process mark the sign of a Knowing 
Organization. The lessons learned resulting from Joint Task Force Exercises should be 
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initiatives on the scale of, “How many strike aircraft do we need to accomplish this 
mission” Instead, they point out that a ship may need a reservist to help sort incoming 
messages. 
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VIII. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

The Navy’s Lessons Learned Program has undergone significant and noteworthy 
changes since its establishment in 1991. Today, the organization fully meets its objective 
of providing the Navy with a “low-cost database system to collate, evaluate, and 
disseminate Navy specific lessons learned.” (OPNAVINST 3500.37B) 

Though it meets its purpose, the author believes there is still room for 
improvement. These improvements can be characterized by either approach or processes, 
with specific suggestions covered in the Recommendations section. 

1. Approach 

As technologies and operations evolve, there is little disagreement with the 
increasingly important need to capture, record, and disseminate lessons learned. However, 
the goal should not be to obtain as many lessons learned as possible. The volume of the 
database is not nearly as important as the quality of the information. The Navy Lesson 
Learned System’s focus should be enabling Knowledge Creation, not Knowledge 
Recording. The ultimate goal should be establishing a Knowledge Management System 
that oversees the creating, securing, capturing, coordinating, combining, retrieving, and 
distribution of knowledge. (Liebowitz, 2000) 

NLLS and Doctrine Department are in a unique position to manage the 
knowledge environment at an organizational level. Learning lessons once is an underlying 
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aim; providing the right information to the right person at the right time is the goal. Many 
organizations, the Navy included, are drowning in information and starving for 
knowledge. Targeted lessons collection, specific knowledge creation, and active 
knowledge dissemination should be the objective of the NLLS system as the Navy enters 
the 21 st century. 

2. Process 

The Navy Lessons Learned System is implementing a Knowledge Attic approach. 
Lessons learned are collected by passive means, validated by active means, but stored and 
disseminated by passive means. The organization does not play an active enough role in 
determining and targeting what it feels are the areas the military most needs to improve. 
Additionally, appropriate lessons learned are not sent to the individuals in the 
organization who would find these lessons to be of the greatest value. Critical elements 
for advancing the organizational intelligence are identifying individual learning, passing 
it on, and creating organizational learning. 

Background research indicates that at almost every level in the organization, 
lessons learned are being collected and disseminated, but to varying degrees and on 
various levels. Whether it is the trouble-shooting procedures developed in the avionics 
shop, or an end-of-cruise report prepared by a fighter squadron, there are varying degrees 
of utility these lessons have to other members in the military. Information Technology 
has reduced the delay previously encountered in collecting and submitting lessons learned 
but has resulted in stove-piping of this information among various units or staffs within 
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the organization. Increased emphasis on networking should be used to bring together 
these multiple bodies of explicit knowledge that exist. By streamlining requirements to 
validate lessons learned and in some cases dropping them all together, NLLS can 
accelerate the process of turning individual learning into organizational learning. 
Observations concerning tactics, techniques, or procedures will still require formalized 
methodology for validation, but there are various other lessons learned that do not. 
Knowledge enablers consisting of Web-based turnover pages or bulletin board discussion 
areas for Operations Officers would foster organizational learning and allow incoming 
units a chance to more quickly assume their operational mode. 

B. RECOMMENDATIONS 

1. Target the User 

NLLS’s most important step should be to undertake a more through survey to 
accurately assess who the customer is. The sample questionnaire identified a lack of 
exposure to the system by most of the respondents. By more precisely identifying the 
customer, NLLS can focus awareness of the system to that population and work to ensure 
the process meets their needs. 

2. Pull the Relevant Information 

The Navy Lessons Learned System relies on a passive collection method. The 
consequence of this is inputs of large volumes of data, from whatever source is available, 
without thought or consideration to defining an overall collection plan. Each observation 
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requires investigation by a Subject Matter Expert regardless of the relevant overall benefit 
to the organization. The end result is a wide-ranging database of discrete facts that took 
many man-hours to complete and is difficult to analyze. Passive collection does have a 
place in NLLS as an enabler of input from all levels of the organization. However, 
targeting the focus of collection would provide greater benefit to the organization as a 
whole. 

Active collection entails a coordinated effort to try to identify solutions to existing 
problems or issues identified by a sponsor. This issue can result from training exercises or 
from shortcomings identified in actual operations. Other sources could be analysts, 
instructors, doctrine writers, or commanders in the field. (Tulak, 1999) The environment 
should be scanned for emerging trends or issues as well as ongoing operations examined 
for possible future ramifications. The Early Bird can provide a good starting point for 
topics currently rising in prominence. (Tulak, 1999) 

Active collection options can range from training evaluators to act as observers to 
identifying and targeting certain learning objectives. Warfare specialists or the Executive 
Steering Committee can identify these learning objectives, and they may change over 
time. Possible examples in the aviation world would be to track Blue on Blue incidents or 
target acquisitions rates during JTFEXs. In the case of target acquisition, metrics would 
be developed to identify reasons the aircrew failed to acquire a target. Data points could 
include weather, equipment failure, or training. From this information, trends would be 
readily identifiable that may help to improve target acquisition or identify shortcomings 
in training syllabi. Measured performance almost always improves. 
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Cost is always an issue, but specific exercises could be targeted for trained 
observers based on their rising importance, emerging trends, or high visibility. JTFEX 00- 
1 provides us with an example. 

Because of the political situation in Puerto Rico, the USS Dwight D. Eisenhower 
Battlegroup was unable to conduct live naval surface fire support (NSFS) training, live 
fire air to ground training, or combined arms training on the Island of Vieques. Efforts 
were made to reschedule, cancel, or move portions of the training to alternate countries 
while on deployment. Lessons learned from this exercise would have been particularly 
valuable for follow-on battlegroups in the event they encountered the same 
circumstances. As important as collecting Eisenhower’s lessons learned would be 
analyzing her trends on deployment due to missed critical training during workups. 

Recent decisions by the governor of Puerto Rico have resulted in the halt of 
training on the Island of Vieques for the USS Enterprise Battlegroup and the USS 
Kearsarge Amphibious Ready Group. With the Enterprise scheduled to deploy in late 
April, Air Wing Commanders and ship Commanding Officers may be scrambling to 
locate alternate training sites to ensure the readiness of their people. Unfortunately, they 
will not be able to gain benefit from the Eisenhower’s experiences.'Because of what some 
people believe was either low submission rates or the poor quality of lessons learned 
generated from JTFEX 00-1, the CINCLANTFLT Fleet Management Site has yet to 
receive this information to include in the database. Trained observers would have helped 
in this case, and the lessons learned could have been used to provide assessment for 
upcoming policy decisions on the future of Vieques. 
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3. 


Push Information to the Users 


A common suggestion to improve NLLS was to increase awareness of the system. 
One method of accomplishing this would be to publish a quarterly newsletter or bulletin. 
Success stories and best practices could be solicited from fleet units, and articles that are 
published should gamer the ship points in the yearly Battle “E” competition. 

An alternative method would be to provide a Top Ten list with a short description 
of the observation for inclusion into already established publications like Fathom or 
Approach. The article would let the reader know where to go to get further information. 
For ships scheduled to undergo JTFEX, or getting ready to go on deployment, briefing 
packages of recent trends noted could be provided. Inputs for these trends could be 
provided by the various SMEs who see the lessons learned on a daily basis and targeted 
by class of ship or area of deployment. 

Finally, unclassified lessons learned should be made available over the Internet to 
those users who have a .mil extension. A link could be posted on the Navy homepage 
along with information about the site. Each of these suggestions may provide incentives 
to use NLLS as well as promote the system to users throughout the fleet. 

4. Categorize by Community 

One difficulty noted while researching the NLLSDB arose from the way in which 
the information was organized. The data is currently grouped under the operations name. 
Noble Shirley, Juniper Stallion, or Fortress Raptor are a few examples. Unless the user is 
familiar with the exercise, extensive time may be spent searching for comparable 
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situations. There is some benefit to this structure; however, an alternate framework for 
arranging or categorizing the information and knowledge should be used that enables 
people to find and use it more effectively. 

A simple sort function that allows the user to group observations by topic or 
community would be helpful. Instead of being grouped by exercise name, they could be 
grouped under Missions with subheadings of Humanitarian Assistance, Noncombatant 
Evacuation, or Foreign Disaster Relief. Operations could have subheadings of Civil, 
Interagency, Multinational, or Combat. Further headings could include sections based on 
Communications, Command and Control, or Intelligence. Organizing the information by 
platform or community is one way to facilitate knowledge flow. 

5. Networking and Resourcing 

The NLLS Organization with the help of the Fleet Management Sites could 
provide a critical role as system intermediaries, responsible for connecting people to the 
information they require. The various Fleet Management Sites already have a strong 
familiarity with how to locate information or expertise in the organization quickly and 
efficiently. Using this experience a Yellow Page Directory, mapping knowledge areas to 
experts within the organization, could be provided. 

C. SUGGESTIONS FOR FURTHER STUDY 

This exploratory study has only begun to uncover the growing role and effect that 
Information Technology will have on Organizational Learning. The explosion of the 
Internet, with all of its benefits and all of its drawbacks, is in its infancy in the military 
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and will continue to amaze and frustrate the men and women who serve on board ships in 
the near future. 

Further areas of study in Organizational Learning would include examining the 
possibility of Web-based turnover pages that would allow the oncoming aircraft carrier, 
as it transits the Atlantic or Pacific Ocean, to take part in the strike planning or shipboard 
operations that will be expected of it once it is on station. Examining the role and 
implications that communities of practice “chat rooms” will have on breaking down 
traditional boundaries and allowing the free flow of information between the organization 
and the environment as they are set up between ships or even battlegroups is another. 
Along with this, a question must be answered: Is lateral communication a threat to good 
order and discipline, blurring the lines between traditional structures and the chain of 
command that is in place? 

Other questions include what sort of expert or decisions support systems the 
Naval Officer of tomorrow will require to deal with the data deluge of information that is 
surely going to take place. Lastly, as our reliance on Information Technology grows, will 
the bandwidth that is available on board ship keep pace, or will it place limitations 
between ships in the battlegroup, creating a digital Darwinism between the haves and the 
have-nots, leaving the smaller ships out of the technological loop of information? 
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APPENDIX A. SURVEY INSTRUMENT 


Navy Lessons Learned System (NLLS) Questionnaire 

[Thank you for agreeing to take time to answers some questions. I am examining the Navy Lessons Learned 
System and its impact on organizational learning. I am conducting this research for my Master Thesis at 
the Naval Postgraduate School. This questionnaire will address three primary topics concerning the Navy 
Lessons Learned system: the submission process, the lessons learned retrieval process, and the effect the 
NLLS system had on the way you planned or approached a scheduled training exercise. The information I 
collect will be strictly confidential and your name will not be used in any way. I appreciate your taking the 
time to fill this out and would welcome any additional comments you have on the system] 


Background 

1. What is your rank and designator? 

2. What positions have you held while on sea duty? 

3. What platforms did you serve on? 

4. Did you make any deployments while onboard, if so how many? 


Lessons Learned Submission 

1. Have you ever heard of the Navy Lessons Learned Center? 

2. Did you know that the Navy Lessons Learned Center has a Web site with an active 
database of all the lessons learned submitted in the last two years? 

3. When your command concluded an exercise were the lessons learned collected and 
forwarded to your ISIC? 

4. During a scheduled exercise or evolution, was lessons learned collection given 
consideration? 

5. Did you consider submitting a NLLS report a worthwhile task? 
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Lessons Learned Retrieval 


1. Did your department have access to the SIPRNET? 

2. Have you visited the NLLS Web or SIPRNET site? 

3. Did you have access to either the classified or unclassified lessons learned CD-ROM? 

4. Have you searched the database for a lesson learned? 

5. What type of lesson learned were you searching for? 

6. On a scale of one to five, where one is of little use, three is of moderate use, and five is 
of exceptional use, rate the usefulness of prior lessons learned as it applied to you in an 
exercise or evolution. 

7. On average how many times a month did you access the lessons learned database? 

8. When your command prepared for an exercise, was the lessons learned database 
consulted during planning? 

9. Did you consider reviewing the NLLS database prior to an exercise a worthwhile task, 
if not why? 

10. What type of circumstance would cause you to reference the lessons learned 
database? 

11. Had referencing the NLLS database resulted in a change in the way you would 
approach tactics, techniques, or procedures (TTP)? 


12. Where would you say you receive the majority of lesson learned information? 


13. Do you have any inputs that you feel would make the NLLS a more effective tool? 
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